scgpt
Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Generative pre-trained models have achieved remarkable success in various domains such as natural language processing and computer vision. Specifically, the combination of large-scale diverse datasets and pre-trained transformers has emerged as a promising approach for developing foundation models. Drawing parallels between linguistic constructs and cellular biology - where texts comprise words, similarly, cells are defined by genes - our study probes the applicability of foundation models to advance cellular biology and genetics research. Utilizing the burgeoning single-cell sequencing data, we have pioneered the construction of a foundation model for single-cell biology, scGPT, which is based on generative pre-trained transformer across a repository of over 33 million cells. Our findings illustrate that scGPT, a generative pre-trained transformer, effectively distills critical biological insights concerning genes and cells. Through the further adaptation of transfer learning, scGPT can be optimized to achieve superior performance across diverse downstream applications. This includes tasks such as cell-type annotation, multi-batch integration, multi-omic integration, genetic perturbation prediction, and gene network inference.
Haotian Cui , Chloe Wang , Hassaan Maan , Kuan Pang , Fengning Luo , Bo Wang ,
Monthly downloads
1000
scGPT ¶
This is the official codebase for scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI.
!UPDATE: We have released several new pretrained scGPT checkpoints. Please see the Pretrained scGPT checkpoints section for more details.
Installation ¶
scGPT is available on PyPI. To install scGPT, run the following command:
$ pip install scgpt
[Optional] We recommend using wandb for logging and visualization.
$ pip install wandb
For developing, we are using the Poetry package manager. To install Poetry, follow the instructions here.
$ git clone this-repo-url
$ cd scGPT
$ poetry install
Note: The flash-attn dependency usually requires specific GPU and CUDA version. If you encounter any issues, please refer to the flash-attn repository for installation instructions. For now, May 2023, we recommend using CUDA 11.7 and flash-attn<1.0.5 due to various issues reported about installing new versions of flash-attn.
Pretrained scGPT Model Zoo ¶
Here is the list of pretrained models. Please find the links for downloading the checkpoint folders. We recommend using the whole-human model for most applications by default. If your fine-tuning dataset shares similar cell type context with the training data of the organ-specific models, these models can usually demonstrate competitive performance as well.
| Model name | Description | Download |
|---|---|---|
| whole-human (recommended) | Pretrained on 33 million normal human cells. | link |
| brain | Pretrained on 13.2 million brain cells. | link |
| blood | Pretrained on 10.3 million blood and bone marrow cells. | link |
| heart | Pretrained on 1.8 million heart cells | link |
| lung | Pretrained on 2.1 million lung cells | link |
| kidney | Pretrained on 814 thousand kidney cells | link |
| pan-cancer | Pretrained on 5.7 million cells of various cancer types | link |
Fine-tune scGPT for scRNA-seq integration ¶
Please see our example code in examples/finetune_integration.py. By default, the script assumes the scGPT checkpoint folder stored in the examples/save directory.
To-do-list ¶
- Upload the pretrained model checkpoint
- Publish to pypi
- Provide the pretraining code with generative attention masking
- Finetuning examples for multi-omics integration, cell type annotation, perturbation prediction, cell generation
- Example code for Gene Regulatory Network analysis
- Documentation website with readthedocs
- Bump up to pytorch 2.0
- New pretraining on larger datasets
- Reference mapping example
- Publish to huggingface model hub
Contributing ¶
We greatly welcome contributions to scGPT. Please submit a pull request if you have any ideas or bug fixes. We also welcome any issues you encounter while using scGPT.
Acknowledgements ¶
We sincerely thank the authors of following open-source projects:
Citing scGPT ¶
@article{cui2023scGPT,
title={scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI},
author={Cui, Haotian and Wang, Chloe and Maan, Hassaan and Pang, Kuan and Luo, Fengning and Wang, Bo},
journal={bioRxiv},
year={2023},
publisher={Cold Spring Harbor Laboratory}
}
Related notebook
BioTuring
Computational methods that model how the gene expression of a cell is influenced by interacting cells are lacking. We present NicheNet, a method that predicts ligand–target links between interacting cells by combining their expression data with prior knowledge of signaling and gene regulatory networks. We applied NicheNet to the tumor and immune cell microenvironment data and demonstrated that NicheNet can infer active ligands and their gene regulatory effects on interacting cells.
BioTuring
Single-cell RNA sequencing (scRNA-seq) data have allowed us to investigate cellular heterogeneity and the kinetics of a biological process. Some studies need to understand how cells change state, and corresponding genes during the process, but it is challenging to track the cell development in scRNA-seq protocols. Therefore, a variety of statistical and computational methods have been proposed for lineage inference (or pseudotemporal ordering) to reconstruct the states of cells according to the developmental process from the measured snapshot data. Specifically, lineage refers to an ordered transition of cellular states, where individual cells represent points along. pseudotime is a one-dimensional variable representing each cell’s transcriptional progression toward the terminal state. Slingshot which is one of the methods suggested for lineage reconstruction and pseudotime inference from single-cell gene expression data. In this notebook, we will illustrate an example workflow for cell lineage and pseudotime inference using Slingshot. The notebook is inspired by Slingshot's vignette and modified to demonstrate how the tool works on BioTuring's platform.
BioTuring
Single-cell RNA data allows cell-cell communications (***CCC***) methods to infer CCC at either the individual cell or cell cluster/cell type level, but physical distances between cells are not preserved Almet, Axel A., et al., (2021). On the other hand, spatial data provides spatial distances between cells, but single-cell or gene resolution is potentially lost. Therefore, integrating two types of data in a proper manner can complement their strengths and limitations, from that improve CCC analysis. In this pipeline, we analyze CCC on Visium data with single-cell data as a reference. The pipeline includes 4 sub-notebooks as following 01-deconvolution: This step involves deconvolution and cell type annotation for Visium data, with cell type information obtained from a relevant single-cell dataset. The deconvolution method is SpatialDWLS which is integrated in Giotto package. 02-giotto: performs spatial based CCC and expression based CCC on Visium data using Giotto method. 03-nichenet: performs spatial based CCC and expression based CCC on Visium data using NicheNet method. 04-visualization: visualizes CCC results obtained from Giotto and NicheNet.